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WHAT IS CLAIMED IS: 

1. An Image processing apparatus which is connected 
to print unit adapted to print an image, and can output 
an image to be printed to the print unit, comprising: 
5 image sensing unit adapted to capture an image of 

a real space; 

generation unit adapted to generate an image of a 
virtual space in accordance with a position and 
orientation of said image sensing unit, which serves as 

10 a viewpoint of the virtual space, on a coordinate 

system shared by the real space and the virtual space; 

storage unit adapted to superpose and storing the 
image of the real space captured by said image sensing 
unit and the image of the virtual space generated by 

15 said generation unit to store a mixed reality space 
image formed by superposing the two images; 

instruction unit adapted to instruct whether or 
not the mixed reality space image stored in said 
storage unit is to be printed; and 

20 output unit adapted to output the mixed reality 

space image stored in said storage unit to the print 
unit upon reception of a print instruction of the mixed 
reality image from said instruction unit. 

25 2. The apparatus according to claim 1, further 
comprising management unit adapted to manage state 
information of the virtual space, and in that said 
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management unit updates the state Information of the 
virtual space In accordance with Input Information or 
at each predeteinnlned timing^ and said generation unit 
generates a virtual space Image according to the 
5 updated state Information of the virtual space. 

3. The apparatus according to claim 2, wherein said 
management unit Is an Independent apparatus connected 
to said Image processing apparatus via a network, and 
the state Information of the virtual space updated by 
said management unit can be received by Image 
processing apparatuses which are equivalent to said 
Image processing apparatus, and are connected to the 
network . 

4- The apparatus according to claim 1, further 
comprising display unit adapted to display the mixed 
reality space Image stored In said storage unit. 

20 5. The apparatus according to claim 1, wherein said 
Image sensing unit Is an HMD which comprises a display 
unit for displaying the mixed reality space Image 
stored In said storage unit, 

said apparatus further comprises a sensor for 

25 measuring a position and orientation near a viewpoint 
position of an observer who wears the HMD on the 
coordinate system, and 



10 
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said generation unit calculates based on the 
position and orientation near the viewpoint position on 
the coordinate system as measurement results of said 
sensor, and a constant which Is measured In advance. 

6. The apparatus according to claim 5, wherein the 
viewpoint includes a first viewpoint and a second 
viewpoint different from the first viewpoint, and 

said generation unit generates an image of the 
virtual space according to a position and orientation 
of the first viewpoint on the coordinate system using 
the position and orientation near the viewpoint 
position on the coordinate system as the position and 
orientation of the first viewpoint, and generates an 
image of the virtual space according to a position and 
orientation of the second viewpoint on the coordinate 
system using a position which is shifted a 
predetermined distance from the position of the first 
viewpoint and the orientation of the first viewpoint as 
the position and orientation of the second viewpoint, 

7 • The apparatus according to claim 1 , wherein said 
image sensing unit is an image sensing device, which 
comprises a display unit for displaying the mixed 
reality space image stored in said storage unit, and is 
attached to a fixed post to be rotatable in a desired 
direction. 
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said apparatus further comprises a sensor for 
measuring a orientation of the attached portion of the 
Image sensing device on the coordinate system, and 

said generation unit uses a measurement result of 
5 said sensor as a orientation of the viewpoint. 

8. An Image processing apparatus, which Is connected 
to a plurality of apparatuses each of which generates a 
mixed reality space Image formed by superimposing 
Images of a virtual space and real space. Is connected 
to print unit adapted to print an Image, and can output 
an Image to be printed to the print unit, comprising: 

selection unit adapted to select a desired one of 
the plurality of apparatuses ; 

Instruction unit adapted to Instruct whether or 
not the mixed reality space Image obtained from the 
apparatus selected by said selection unit Is to be 
printed; and 

output unit adapted to output the mixed reality 
space Image to the print unit upon reception of a print 
Instruction of the mixed reality Image from said 
Instruction unit. 

9. The apparatus according to claim 8, further 
25 comprising generation unit adapted to generate one 

Image which Includes the mixed reality space image 
obtained from the apparatus selected by said selection 
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unit, and in that said output unit outputs the one 
Image to the print unit upon reception of the print 
instruction from said instruction unit. 

5 10. The apparatus according to claim 9, wherein when 
a plurality of mixed reality space Images are obtained 
from the apparatus selected by said selection unit, 
said generation unit reduces the mixed reality space 
Images and generates one image which includes the 
10 reduced mixed reality space images. 

11. The apparatus according to claim 9, further 
comprising display unit adapted to display the one 
image generated by said generation unit. 

15 

12. An image processing method to be executed by an 
image processing apparatus which is connected to print 
unit adapted to print an image, and can output an image 
to be printed to the print unit, comprising: 

20 an image sensing step of capturing an image of a 

real space using image sensing unit; 

a generation step of generating an Image of a 
virtual space in accordance with a position and 
orientation of the image sensing unit , which serves as , 
25 a viewpoint of the virtual space, on a coordinate 

system shared by the real space and the virtual space; 
a storage step of superposing and storing the 
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Image of the real space captured In the Image sensing 
step and the Image of the virtual space generated in 
the generation step to store a mixed reality space 
'image formed by superposing the two images in a memoory; 
5 an instruction step of instructing whether or not 

the mixed reality space image stored in the memory is 
to be printed; and 

an output step of outputting the mixed reality 
space image stored in the memory to the print unit upon 
10 reception of a print instruction of the mixed reality 
image from the instruction step. 

13* An image processing method to be executed by an 
image processing apparatus, which is connected to a 

15 plurality of apparatuses each of which generates a 
mixed reality space image formed by superimposing 
images of a virtual space and real space, is connected 
to print unit for printing an image, and can output an 
image to be printed to the print unit, comprising: 

20 a selection step of selecting a desired one of 

the plurality of apparatuses; 

an instruction step of instructing whether or not 
the mixed reality space image obtained from the 
apparatus selected in the selection step is to be 

25 printed; and 

an output step of outputting the mixed reality 
space image to the print unit upon reception of a print 
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Instruction of the mixed reality Image from the 
Instruction step. 

14 . A computer program making a computer function as 
an Image processing apparatus of claim !• 

15. A computer program making a computer execute an 
Image processing method of claim 12. 

16. A computer readable storage medium storing a 
computer program of claim 14. 

17. A computer readable storage medium storing a 
computer program of claim 15. 

18. An Image processing apparatus which Is connected 
to print unit adapted to print an Image, and can output 
an Image to be printed to the print unit, comprising: 

Image sensing unit adapted to capture a real 
space Image; 

generation unit adapted to generate a virtual 
space Image according to a viewpoint position and 
orientation on a coordinate system shared by real and 
virtual spaces to have said image sensing unit as a 
viewpoint on the virtual space; 

composition unit adapted to superimpose the real 
space image captured by said image sensing unit and the 
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virtual space Image generated by said generation unit; 

display unit adapted to display a mixed reality 
space image obtained by said composition unit; and 

instruction unit adapted to give, to the print 
5 unit, an output instruction of the mixed reality space 
image composited by said composition unit, and 

in that said generation unit has a plurality of 
calculation methods and sequences for generating the 
virtual space image, and 
10 when the instruction is received from said 

instruction unit, said generation unit re-generates a 
virtual space image using a second calculation method 
and sequence, which are different from a first 
calculation method and sequence used to generate the 
15 virtual space image, said composition unit 

re-composites a mixed reality space image from the 
re -generated virtual space image and the real space 
image, and the print unit prints the re-composited 
mixed reality space image on a medium. 

20 

19- An image processing apparatus which is connected 
to print unit adapted to print an image, and can output 
an image to be printed to the print unit, comprising: 

image sensing unit adapted to capture a real 
25 space image; 

generation unit adapted to generate a virtual 
space image according to a viewpoint position and 
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orientation on a coordinate system shared by real and 
virtual spaces to have said image sensing unit as a 
viewpoint on the virtual space; 

composition unit adapted to superimpose the real 
space image captured by said image sensing unit and the 
virtual space image generated by said generation unit; 

display unit adapted to display a mixed reality 
space image obtained by said composition unit; and 

instruction unit adapted to give, to the print 
unit, an output instruction of the mixed reality space 
image composited by said composition unit, and 

in that said generation unit has a plurality of 
data sets for generating the virtual space image, and 

when the instruction is received from said 
instruction unit, said generation unit re«generates a 
virtual space image using a second data set, which is 
different from a first data set used to generate the 
virtual space image, said composition unit 
re-composites a mixed reality space image from the 
re -generated virtual space image and the real space 
image, and the print unit prints the re-composited 
mixed reality space image on a medium. 

20. The apparatus according to claim 18, wherein a 
pixel size of one side of the virtual space image 
generated by said generation unit assumes an arbitrary 
value which has, as an upper limit, a product of a 
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length of one side and a resolution of an Image that 
the print unit can output . 

21. The apparatus according to claim 18, further 
5 comprising management unit adapted to manage state 

Information of the real and virtual spaces, and In that 
said management unit updates the state Information of 
the virtual space In accordance with Input Information 
or at each predetermined timing, and said generation 
10 unit successively generates virtual space Images In 
accordance with the updated state Information of the 
real and virtual spaces. 

22. The apparatus according to claim 21, wherein said 
15 management unit is an independent apparatus connected 

to said image processing apparatus via a network, and 
the state information of the virtual space updated by 
said management unit can be received by image 
processing apparatuses which are equivalent to said 
20 image processing apparatus, and are connected to the 
network . 

23. The apparatus according to claim 18, wherein said 
apparatus can be simultaneously used by at least one 

25 user, and has said image sensing unit, said generation 
unit, said composition unit, said display unit, and 
detection unit adapted to detect a viewpoint position 
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and visual axis direction of a user, wlilch are prepared 
for each user. 

24 • The apparatus according to claim 23, wherein said 
5 display unit, or said display unit and said Image 
sensing unit, forms or form a head-mounted display 
device to be mounted on a head of the user. 

25. A computer program making a computer function as 
10 an Image processing apparatus of claim 18. 

26. A computer readable storage medium storing a 
computer program of claim 25. 

15 27. An Image processing apparatus which Is connected 
to print unit adapted to print an Image, and can output 
an Image to be printed to the print unit, comprising: 

Image sensing unit adapted to capture a real 
space image ; 

20 generation unit adapted to generate a virtual 

space image according to a viewpoint position and 
orientation on a coordinate system shared by real and 
virtual spaces to have said image sensing unit as a 
viewpoint on the virtual space; 

25 composition unit adapted to superimpose the real 

space image captured by said image sensing unit and the 
virtual space image generated by said generation unit; 
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display unit adapted to display a mixed reality 
space Image obtained by said composition unit; 

Instruction unit adapted to give, to the print 
unit, an output Instruction o£ the mixed reality space 
5 Image composited by said composition unit; and 

determination unit adapted to check a positional 
relationship among said Image sensing unit, a real 
space object, and a virtual space object on the 
coordinate system, and determining whether or not the 
10 real space object Is occluded by the virtual space 
object In the mixed reality space Image, and 

In that said generation unit re-generates a 
virtual space Image using a second calculation method 
and sequence, which are different from a first 
15 calculation method and sequence used to generate the 
virtual space Image, In accordance with a result of 
said determination unit, said composition unit 
re -composites a mixed reality space Image from the 
re -generated virtual space Image and the real space 
20 Image, and the print unit prints the re-composlted 
mixed reality space Image on a medium. 

28. The apparatus according to claim 27, further 
comprising management unit adapted to manage state 
25 Information of the real and virtual spaces, and In that 
said management unit updates the update Infoannatlon of 
the virtual space In accordance with Input Information 
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or at each predetermined timing, and said generation 
unit successively generates virtual space Images In 
accordance wltli the updated state Information of the 
real and virtual spaces. 

5 

29. The apparatus according to claim 28, wherein said 
management unit Is an Independent apparatus connected 
to said Image processing apparatus via a network, and 
the state Information of the virtual space updated by 
10 said management unit can be received by Image 

processing apparatuses which are equivalent to said 
Image processing apparatus, and are connected to the 
network . 

15 30. The apparatus according to claim 27, wherein said 
apparatus can be simultaneously used by at least one 
observer, and has said Image sensing unit, said 
generation unit, said composition unit, said display 
unit, and detection unit adapted to detect a viewpoint 

20 position and visual axis direction of an observer, 
which are prepared for each observer. 

31. The apparatus according to claim 30, wherein said 
display unit, or said display unit and said image 
25 sensing unit, forms or form a head-mounted display 
device to be mounted on a head of the user. 
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32. A computer program making a computer function as 
an Image processing apparatus of claim 21. 

33. A computer readable storage medium storing a 
5 computer program of claim 32. 
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